Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WIP: Add boundary comm inner loop specializations for performance and allow setting vector length #1175

Open
wants to merge 9 commits into
base: develop
Choose a base branch
from
Open

WIP: Add boundary comm inner loop specializations for performance and allow setting vector length #1175

wants to merge 9 commits into from
+169 −63

Conversation

lroberts36
Copy link
Collaborator

@lroberts36 lroberts36 commented Sep 12, 2024 •
edited
Loading

PR Summary

This PR adds various inner loop specializations to avoid calling reductions, coordinate transformations, and masks in the inner boundary buffer setting loops where possible. I found this made a little bit of a difference for the multigrid communication, not sure that it really matters elsewhere.

It also exposes a flag parthenon/mesh/comm_vector_length which sets the vector length of the outer team policy. We have always used the Kokkos default value here (which is one). I am not sure if changing this value has any impact, but I at least wanted to be able to play around with it.

As an aside, once #1142 is in, we may want to switch these loops to par_fors instead of raw Kokkos loops.

PR Checklist

  • Code passes cpplint
  • New features are documented.
  • Adds a test for any bugs fixed. Adds tests for new features.
  • Code is formatted
  • Changes are summarized in CHANGELOG.md
  • Change is breaking (API, behavior, ...)
    • Change is additionally added to CHANGELOG.md in the breaking section
    • PR is marked as breaking
    • Short summary API changes at the top of the PR (plus optionally with an automated update/fix script)
  • CI has been triggered on Darwin for performance regression tests.
  • Docs build
  • (@lanl.gov employees) Update copyright on changed files

Yurlungur
Yurlungur approved these changes Sep 12, 2024
View reviewed changes
Copy link
Collaborator

@Yurlungur Yurlungur left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! This is a good change that should hopefully help with performance.

src/mesh/mesh.cpp Show resolved Hide resolved
@lroberts36
Copy link
Collaborator Author

This PR is out of date with @bprather changes in #1196, as well as some other changes. It probably is still worth including at some point, but the required changes are pretty simple to just put in a new PR.

@lroberts36 lroberts36 changed the title Add boundary comm inner loop specializations for performance and allow setting vector length WIP: Add boundary comm inner loop specializations for performance and allow setting vector length Nov 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Yurlungur Yurlungur Yurlungur approved these changes

@pdmullen pdmullen pdmullen approved these changes

@pgrete pgrete Awaiting requested review from pgrete

@bprather bprather Awaiting requested review from bprather

@jdolence jdolence Awaiting requested review from jdolence

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@lroberts36 @Yurlungur @pdmullen